Seeing Virtual Objects: Simulating Reflective Surfaces on Emissive Displays

In this paper, we work toward creating display systems that can present virtual proofs and replicas that behave like real physical surfaces in the actual lighting environment surrounding the display. The goal is to recreate the experience of directly viewing a reflective surface while using self-luminous display screens. We use a computational approach that models the physical light sources and actively tracks the observer to generate simulated reflections that are consistent with the observer’s real viewing position and the real lighting present. The spectral composition of the ambient illumination is interactively sensed and the surface colors are calculated with a real-time multispectral rendering pipeline to allow the rendered model to automatically update its color with changes in the real lighting. Introduction Computers, presenting image content on display screens, are widely used in applications where the intention is to convey the appearance of object surfaces. Computer-based displays provide the means to rapidly evaluate the appearance of different options in proofing and material design applications, allow for the presentation of stimuli that would be difficult to construct for studies of material appearance, and support widespread dissemination of culturally significant objects, such as artwork, that otherwise may not be easily accessible. In these cases, a primary goal is for the appearance of the reproduction onscreen to represent the appearance of the physical object surface that is being portrayed. In this paper, we present a framework for displaying surfaces on self-luminous display screens that begins to recreate the experience of directly viewing a real physical reflective surface. When viewing a real object surface, the patterns of light that ultimately reach the observer are a product of the light sources in the environment, the material properties of the surface, and the position of the observer. The appearance of the object changes with variations in the spectral composition or geometry of the incident light and also with changes in the geometric relationship between the illumination, surface, and observer. In contrast, display screens are typically viewed in a dim surround and present objects as part of a self-luminous image. The object in the image or interactive simulation is typically located in a virtual space separated from the viewer and has its own virtual illumination that is independent of the real illumination in the observer’s environment. In this paper, we work toward creating a display system (shown in Figure 1) that can present virtual proofs and replicas that behave like real physical surfaces in the actual lighting environment surrounding the display. This concept is achieved with traditional self-luminous display hardware using a computational approach that models the physical light sources and actively tracks the observer to generate simulated reflections that are consistent with the observer’s real viewing position and the real lighting present. This work is related to research on exact color soft proofing systems, where the light output of the screen is matched in absolute colorimetry to the diffuse surface colors of a physical hardcopy [1]. By incorporating aspects from realistic image synthesis [2], spatial augmented reality [3], and light-sensitive displays [4], our goal is to extend this paradigm to support a range of material surfaces. We incorporate texture and gloss properties by accounting for the geometric configuration of the illumination relative to the screen and using computer-graphics rendering methods to light the surface. For a virtual object to behave like a real surface, the virtual reflections need to update to remain consistent with changes in the geometry or the spectral composition of the illumination. The system accounts for the position of the viewer relative to the screen and the angle of the screen with respect to the real lights, which allows the luminance of diffuse reflections and patterns of specular reflections to change as they would for a real physical surface. Finally, by interactively monitoring the spectral composition of the ambient light and incorporating a real-time multispectral color rendering pipeline, the chromaticities of virtual surface reflections are able to change automatically to remain consistent with the real physical lighting present. If a reproduction on an electronic display can sufficiently imitate the behavior of a real-world surface that is perceived as an illuminated reflective object, it can provide the advantages of a digital proof, while still retaining the object-mode appearance attributes of the physical surface it is attempting to portray. Though our current system still has limitations, we take initial steps toward this goal. In the following sections, we describe background and related work, present a methodology for modeling and recreating the appearance of reflective surfaces, and present an initial prototype system along with a description of its capabilities. Figure 1. Left, a real physical painting and right, a virtual model created from captured data [5] that has been interactively rendered and displayed on a selfluminous display screen. Background and Related Work The creation of a system to reproduce the appearance of objects in real physical space has a basis in computer-based proofing, which provides methodologies for reproducing color and appearance attributes, as well as systems for spatially-augmented reality [3] that work to bridge the gap between the virtual and physical world. Laihanen [6] developed an early system for exact soft proofing, attempting to produce an appearance match by reproducing the exact colorimetry (chromaticity and absolute luminance) of prints on a CRT screen. Recently, Hill [1] developed a display system for exact color proofing to allow for direct comparisons of hardcopy and soft-proof patches on an LCD screen in an illuminated light booth. Hill’s system was calibrated to reproduce the exact colorimetry of a physical ColorChecker, using spectral data on the light sources and by adjusting the luminance output level of screen regions until they matched a physical mask placed on the screen. Research efforts have also investigated incorporating gloss properties into the proofing process. Gatt et al. [7] performed goniometric measurements to develop a predictive BRDF model for the gloss properties of printed materials. Patil et al. [8] developed a gloss soft-proofing system that generated simulated prints in a virtual environment by mapping the images to 3D planes and allowed the user to change the virtual viewpoint using QuickTime VR. The tangiBook system [9] used tracking information on the orientation of the screen and the real position of the observer to update the virtual reflections shown on a display screen, to provide natural forms of interactivity with virtual surfaces that had color, gloss and texture properties. The tangiBook system provided the capability to render virtual objects within different virtual lighting environments, but did not attempt to maintain spectral or geometric consistency with the real physical lighting environment. The lighting-sensitive display [4] introduced the concept of illuminating virtual content on a display according to the lighting in the real environment surrounding the screen. This system interactively acquired an image map of the lighting in the environment with a camera and used this information to relight a static 3D scene using image-based rendering methods. Koike and Naemura [10] developed a display system capable of directionally modulating the light output using a lenticular array to simulate the view-dependent properties of surface BRDF. Fuchs et al. [11] explored different configurations of light field display systems capable of responding to spatial or directional incident light and outputting spatial or directional patterns of light. In this work, we bring together concepts from colorimetricbased proofing systems, realistic image synthesis, and lighting sensitive displays in an attempt to create virtual objects under computer control that appear like the real physical objects they are portraying. System Overview The display system in this paper is designed to recreate colorimetrically the patterns of reflected light that a physical surface, positioned at the screen’s location in the real lighting environment, would produce in the direction of the observer. Toward this goal, the system combines sensing and modeling of the real-world lighting present, tracking technologies, and a multispectral real-time rendering engine to simulate the types of light-surface interactions that contribute to the appearance attributes of a surface (color, gloss, and texture). The results are displayed through a photometrically-calibrated screen so that the luminance and chromaticity of the emitted light can be matched to the light that would be reflected by an object at the screen’s position. Object Surface Modeling The properties of the virtual surfaces, including diffuse color, are represented in manner that is independent of illumination so that the virtual output can be interactively updated as lighting changes, as a physical sample would change with the lighting. Diffuse Color Color calculations are performed using a multispectral factor methodology [12] by multiplying the coefficients of reflectance and illumination over a set of six optimized spectral channels developed for real-time multispectral rendering [13]. The diffuse color data for the virtual surfaces are represented in a multispectral form similar to reflectance factor, where the reflectance coefficient for each wavelength band varies between 0 and 1. To support spatially varying color for objects such as paintings or digital print proofs, six-channel reflectance data are maintained as two threechannel (RGB) floating point images. Gloss The gloss properties of the surface are represented using the specular reflectance parameters of the Ward BRDF model [14]. The Ward α (specular roughness) parameter is used to describe the width of the specular lobe. The parameter describing the magnitude of the specular reflectance, ρs, may be specified for each of the six multispectral channels to allow for spectrallyselective front surface reflection. Small Scale Geometry and Texture The system is intended to display surfaces with principally planar geometry, like the real screen has, so that the locations of virtual reflections are consistent with the physical location of light emitted from the screen. To support object surfaces with some texture or relief, small scale geometry (surface height << viewing distance) is modeled as a dense height field relative to the physical plane of the screen. The effect of this geometry on the orientation of surface facets, relative to the planar screen surface, is represented with image-based normal maps [15]. The selfshadowing that results from higher surface points blocking light from reaching other surface points is represented with image-based horizon shadow maps [16]. Modeling the Light Booth Illumination The system uses a model of the real-world illumination in the screen’s environment to allow the virtual surface to be rendered in a manner consistent with a physical surface at the same position. For the lighting environment, we selected a light booth that has tungsten, fluorescent D50-simulating, and D65-simulating light sources. Figure 2. Images captured of the spatial luminance variation for the three lighting options: (A) tungsten, (B) a fluorescent D50 simulator, and (C) a D65